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ABSTRACT 

In recent years there has been a growing concern with the quality of instruction in higher education and how 
the instructional process might be improved while institutions are experiencing many fiscal constraints. One 
particular concern has been the rising number of adjunct faculty teaching courses at universities and the potential 
impacts on instructional programs. The purpose of this study was to examine student course evaluations of classes 
within the College of Education at East Tennessee State University (ETSU) and identify relationships and 
differences in the perceived levels of instructional quality based on faculty characteristics. This provides for the 
identification of characteristics most strongly associated with high ratings of instructional performance, as measured 
along the dimensions of attitude, methods, content, interest, and instructor characteristics. These data allow 
comparisons with other institutions of higher learning. Both undergraduate and graduate students within the College 
of Education were given the Student Assessment of Instruction (SAI) instrument for course evaluation. Data for the 
years 1992 through 1999 were used in this analysis. The five SAI subscales (attitude, methods, content, interest, and 
instructor) comprise the total score on the in strum ait. Analysis of variance and t-tests were used to identify 
differences on die SAI subscales based on instructor rank, department, year of administration, and semester. 

Analysis of the data revealed differences on the total rating scale (F=5.601, p < .05) and on each subscale. Scheflfe’s 
Post Hoc Multiple Comparisons Tests indicated that those at the rank of instructor received higher ratings than full 
professors. Adjunct faculty ratings were similar to those of full professors. The differences in the ratings between 
faculty members at different ranks varied considerably, however, between departments. There were significant 
overall differences in ratings between the different academic departments, although some of these differences 
appeared to be related to whether the courses being rated were undergraduate or graduate level. No significant 
differences were found between ratings during the fall and spring semesters, between male and female faculty, or 
across the seven years of administration. Differences in ratings may be due to factors other than instructional 
delivery, which may call for further exploration. 
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INTRODUCTION 



The improvement of instructional delivery has been of paramount concern in higher education during recent 
years. One particular concern has been die rising number of adjunct faculty teaching courses at universities and the 
potential impacts on instructional programs. If a provider has devoted resources to a particular educational activity, 
it wants to make sure that the information and skills needed are delivered. This is helpful both for future funding 
and further provision of educational services and for current satisfaction levels. From the consumer’s standpoint, it 
is important to actually receive the information or skills that the educational activity has been promoted to provide. 
For practitioners in the field of adult or continuing education, the provision of information and skills is important for 
die involved learners’ career goals, personal growth, and technological literacy. This is nowhere more pertinent than 
on the college campus, particularly with graduate students, who are usually adult learners. Most colleges and 
universities have developed some sort of assessment to measure the effectiveness of methods of instruction, course 
content, particular faculty members, and curriculum. 

At East Tennessee State University (ETSU), die Student Assessment of Instruction (SAI) course evaluation 
form, developed by Lawrence Aleamoni, has been used since the fall of 1992. At that time, the Faculty Senate 
Subcommittee on SAI recommended this form to the Faculty Senate. The use of the SAI was subsequently endorsed 
by the ETSU Faculty Senate and approved for use by the Vice-President for Academic Affairs. Another form 
developed internally had been used previously. East Tennessee State University is a regional state-supported 
university located in Johnson City, TN. The student enrollment is approximately 10,000. The data analyzed in this 
study were drawn from the College of Education, which includes four academic departments and offers bachelor s, 
masters, and doctoral degrees. The four departments are 1 ) Curriculum and Instruction (CUAI), 2) Educational 
Leadership & Policy Analysis (ELPA), 3) Human Development and Learning (HDAL), and 4) Physical Education, 
Exercise, and Sports Science (PEXS). Each of the departments offers both bachelor's and master's degree. In 
addition, ELPA offers the Educational Specialist and Doctor of Education degrees. The issues faced at this 
institution are similar to those at other institutions of similar size and mission. 

The purpose of this data analysis project was to investigate differences in student ratings of instructional 
effectiveness based on the demographic and academic characteristics of instructors in the College of Education at 
this state-supported university. It was anticipated that die results of this study would shed light on those factors that 
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are most closely related to instructional effectiveness, as measured on the SAI, and point to areas in which the 
institution might improve current practice. 

Review of Related Literature 

The validity of student evaluations of teaching (SETs) has long been of concern to researchers. Including a 
peak of studying validity of student ratings in the years of 1976-80, researchers have questioned bias in SETs 
through the years (Green wald, 1997). More recent literature suggests, however, that the primary questions having to 
do with ratings validity in the 1970s were largely answered and effectively put to rest by later studies. Grades and 
ratings were found to be correlated but not necessarily in such a way as to create a contaminated interpretation. 
Current research still addresses concerns having to do with conceptual structure, convergent validity, discriminant 
validity, and consequential validity (Greenwald, 1997). 

The complexity of the instructional process has also been considered in looking at SETs. Marsh and Roche 
(1997) critiqued the tendency to ‘"puree” the various distinct components of students’ ratings into one mixture, rather 
than to treat them as the “apples and oranges” that make up effective teaching. Questions of validity and usefulness 
of SETs have to take into account the multidimensionality of teaching. 

d’Apollonia and Abrami (1997) found that although effective teaching may be multidimensional, student 
ratings of instruction measure general instructional skill. This general skill can be broken down into three subskills: 
delivering instruction, facilitating interactions, and evaluating student learning. Student ratings are moderately 
valid, but administrative, course, and instructor characteristics can influence them. However, d’Apollonia and 
Abrami cautioned against rigidly controlling for biasing variables and remarked, “We recommend that student 
ratings not be over- interpreted” (p. 1205). 

Proper measurement must be considered in order to accurately interpret overall instructor ratings. Specifically, 
questions have been raised about whether individual level data or class aggregate statistics should be used in 
assessing instruction. Cranton and Smith (1990) empirically demonstrated that a clearly different structure of 
student ratings of instruction is produced by class means when compared with individual ratings and deviations from 
class means. A small but significant relationship was found between course characteristics and perceptions of 
instruction, varying with the unit of analysis used. This research supported the trend to focus in on class means as 
an appropriate unit of analysis. The proper use of student ratings has also been debated. Seldin (1993) cautioned 



that student ratings should never be the only basis for evaluating teaching effectiveness. Small differences between 
faculty members’ scores should not be misinterpreted. 

The increasing use of technology in “classroom” instruction poses new questions in evaluating teaching 
effectiveness. For example, Spooner et al. (1999) found that student ratings in two courses offered both on campus 
and off campus using different methods of instruction (including electronic media) showed no significant difference. 
Course, instructor, teaching, and communication ratings were all similar across courses and settings. More research 
is called for on this issue. 

The changing profile of faculty in academia leads toward speculation on the impact of faculty characteristics 
on instructional effectiveness. Specifically, the increasing use of part-time faculty in higher education has been a 
trend in recent years. Research completed by Burgess and Samuels (1999) in a large urban multi-campus 
community college district showed that students who had been taught by part-time (PT) faculty and then full-time 
(FT) faculty in sequential courses were less likely to complete or get a grade of “C” or better than students in other 
instructor status combinations (i.e., PT-PT, FT-FT, or FT-PT). These data seem to indicate that part-time faculty 
under-prepare their students for following courses taught by fiill-time instructors. Issues having to do with pay, job 
security, benefits, and facilities may have influenced the findings of this study. The 1993 National Survey of 
Postsecondary Faculty found that 47.8 percent of part-time faculty took their jobs because full-time work was not 
available. The reluctant part-timers showed greater proportions of women and part-timers under 35 than did the 
willing part-timer group (Palmer, 1999). 

How does this translate into four-year public institutions? A 1994 report showed that four-year public colleges 
and universities are less likely than two-year institutions to use part-time instructors. However, there was still a 
substantial proportion of part-time faculty in universities. If Teaching Assistants (TAs) and Research Assistants 
(RAs) were included in the overall part-time numbers, around 47.5% of faculty in 4-year public institutions were 
part-time. If TAs and RAs were excluded, the percent was 23.6% (Zimbler, 1994). 

The question of gender differences in instructional effectiveness has also been raised. The number of women 
faculty teaching on a part-time basis nearly tripled from 1976 to 1995. However, the number of newly hired full- 
time women increased by 55% over the 20 years. The great increase in the number of female faculty and the 
moderate increase in male faculty caused a change in the proportions of men and women faculty members. Female 
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faculty members increased from 27 to 39 percent, whereas women full-time faculty only increased from 17 to 20%. 
About 48% of women faculty worked part-time in 1995, compared to 36% of men. Other research has shown that 
the increasing number of women in academia is slowly eroding the dominance of white males in faculty positions 
(Clery, 1998). It is with this eye on the changing demographics of faculty in higher education and the issues 
associated with student ratings of instruction that the research presented in this paper was undertaken. 

Research Questions 

In an effort to examine differences in the ratings of instructional effectiveness based on the demographic and 
academic characteristics of instructors, six primary research questions were addressed: 1 ) What is the profile of the 
faculty in the College of Education at ETSU, and how has it changed between 1 992 and 1999 ? 2) Are there 
differences in student ratings of instruction between faculty members of different rank? 3) Are there significant 
differences in the ratings of instruction completed in the Fall and Spring Semesters? 4) Are there differences in 
student ratings of male and female faculty members? 5) Are there differences in the ratings of instruction among 
the different departments? 6) Are there differences in student ratings of instruction across the different years of 
administration? 

METHODS 

Source of the Data and Methods of Data Collection 

The data were obtained from the Office of Institutional Effectiveness and Planning at East Tennessee State 
University. The data are from information obtained on course evaluations at the end of each semester. Students 
fill out forms called Student Assessments of Instruction (SAI), which are then be machine scored. A Course 
Identification Form is completed by the assessment proctor. This form provides descriptive information about the 
course. Client Support Services in the Office of Information Technology scans the forms and builds a data file. A 
SAS program was then run to compile the data with which this project was begun. This file was then imported into 
the SPSS program for analysis. The data file contained information from Fall 1992 thru Spring 1999. During each 
Fall and Spring Semester full-time faculty members in the College of Education must be evaluated in a minimum of 
two classes. Adjunct/part-time faculty are usually evaluated in every class. 
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The answers to the questions on the evaluation are coded as follows: “Agree Strongly”=4, “Agree”=3, 
“Disagree”=2, and “Disagree Strongly”=l, except where the questions are worded negatively. High numbers are 
associated with positive responses. Answers are recoded on negatively worded questions, so that high values are 
associated with positive responses. 

There are five subscales used in the analysis of the data, made up of various combinations of the questions on 

the SAL The subscales and total scale are computed as follows: 

Attitude: Q1 + Q6 + Q15 + Q21 

Methods: Q2 + Q7 + Q12 + Q18 

Content: Q4 + Q8 + Q13 + Q16 

Interest: Q5 + Q10 + Q17 + Q19 

Instructor: Q3 + Q9 + Q1 1 + Q14 + Q20 

Total: Attitude + Methods + Content + Interest + Instructor. 

RESULTS 

The results are organized around the six major research questions. Each of the questions is examined 
sequentially. Since the tables were quite large they have all been placed in Appendix A. 

Research Question #1: What Is the profile of the faculty In the College of Education at ETSU, and how has it 
changed between 1992 and 1999 ? 

A descriptive profile was obtained of the faculty in the College of Education. The characteristics of the group 
are shown in Table 1. As noted in Table 1, there is a fairly even distribution of male and female faculty members. 
However, the percent of female faculty member ratings has increased from 1992 (46.3%) to 1999 (532%). In fact, 
the percentage of male and female faculty member ratings “switched” over the seven year period, with females 
being in the “majority” in 1999. 

The largest numbers of faculty are in the “Adjunct/Part-Time” category, with the other larger numbers being 
in the ‘"Professor” and “Assistant Professor” categories. The number of faculty members was smaller in 1 992 and 
1999 since data were available for only one semester in each of those years. 

Consistent with the literature, there was a substantial change in the use of adjunct faculty over the seven-year 
time span. The number of adjunct faculty ratings increased from 17.7% of all ratings in 1992 to 39.9% in 1999. 
During this period there was a concurrent decrease in the number of ratings of full professors from 24.5% to 17.6%. 
A smaller number of faculty members are represented in the Educational Leadership and Policy Analysis (ELPA) 
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department than in the other three academic departments in the College of Education. The ELP A department offers 
only graduate degrees, whereas the other departments also provide undergraduate courses. 



Research Ouestf on #2; Are there differences in student ratings of instruction between faculty members of 
different rank? 

An Analysis of Variance was completed to determine if there were significant differences in the student ratings 
of instruction for all of the subscale means and the total mean, by rank. As shown in Table 2, all of the scales 
showed a significant difference between the ranks of faculty, for both the total sample and the departmental 
subgroups. A series of Scheffe’s Post Hoc Multiple Comparisons Tests gave more information about the exact 
nature of these differences. Tables 3 through 8 show the details of which ratings were significantly higher or lower 
for particular ranks, for both the total sample and the departmental subgroups. 

In terms of the total sample, instructors had significantly higher ratings than full professors on the methods, 
content, and interest subscales, but no significant differences were found between these two groups on the attitude 
subscale and the total scale. Professors did score higher than instructors on the instructor subscale. Assistant 
professors had significantly lower ratings than instructors on the total scale and on all the subscales except the 
instructor subscale. Adjunct/part-time instructors scored significantly higher than assistant professors on the 
methods, content, and interest subscales and the total scale. Associate professors scored lower than instructors on 
the methods, content, and interest subscales. When considering the total sample, no significant differences were 
found between the professor and the adjunct/part-time ratings. 

The departmental comparisons, however, indicated some very different patterns. For example, in CUAI, 
senior faculty appeared to score higher than junior faculty. Professors scored higher than assistant professors on all 
five subscales and the total scale. This same pattern was evident in ELPA, where professors were rated higher than 
associate professors on the attitude and interest subscales and the total scale. ELPA professors were rated higher 
than adjunct faculty on all five subscales and the total scale. A very different pattern can be seen in PEXS, where 
professors are rated lower than all the other ranks on the attitude and content dimensions, and lower than all but 
associate professors on the remaining subscales. These differences might be due to the type of classes that are 
offered in the different departments. For example, ELPA only offers graduate programs and has a very different 
clientele than PEXS, where many students from across campus enroll in 1000 and 2000 level activity courses. 
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In an effort to determine if such departmental differences might be due to the nature of the courses being 
taught, an additional analysis was run to see if there were differences in the mean ratings of graduate and 
undergraduate courses. These results are shown in Table 9. As shown in Table 9, when the total sample was 
examined, graduate and undergraduate differences were evident on all the subscales except the method subscale. 
There were also differences on the total scale. In all cases, the ratings in graduate courses were higher. Within 
CUAI there were no significant differences on any of the subscales except the instructor subscale, where instructors 
in graduate courses were rated higher. In contrast, in HDAL all of the graduate- to- under graduate comparisons were 
significant, with ratings being higher in the graduate courses. Finally, in PEXS, there were significant differences 
on all of the subscales and on the total scale. However, in each case the undergraduate ratings were higher. 

Clearly, differences between graduate and undergraduate ratings vary greatly by department. 



Research Question #3: Are there significant differences in the ratings of instruction completed in Fall and 
Spring Semesters ? 

A series of t-tests for Independent Means was performed to determine if there were significant differences in 
the total and subscale means between the fall and spring semesters. As shown in Table 10, no significant differences 
were found. The scheduling of academic coursework does not appear to influence the ratings of instruction. 

Research Question #4: Are there differences in student ratings of male and female faculty members ? 

A series of t-tests for Independent Means were also completed to determine if there were significant 
differences in the ratings of faculty members according to gender. The data are presented in Table 1 1 . There were 
no significant differences found in any of the subscales or the total means of the student ratings, according to faculty 
gender. 



Research Question #5: Are there differences in the ratings of instruction among the different departments? 

An Analysis of Variance was run to test the significance of differences among the ratings received by courses 
in different departments within the College of Education. The four departments are Educational Leadership and 
Policy Analysis (ELPA), Curriculum and Instruction (CUAI), Human Development and Learning (HDAL), and 
Physical Education, Exercise, and Sports Science (PEXS). As shown in Table 12, significant differences were 
found on all of the subscales and total means. Scheflfe’s Post Hoc Multiple Comparisons test placed the departments 
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in different homogeneous subsets. The ELPA department was in the highest-rated subset on all of the subscales and 
the total means. The CUAI department overlapped the ELPA department in the higher subset on the attitude and 
interest subscales. The PEXS department overlapped with the ELPA subset on the method subscale. On the interest 
and instructor subscales, three homogeneous subsets were created. The PEXS and the CUAI departments were in 
the middle group on the interest subscale, whereas the HDAL and the CUAI departments were in the middle group 
on the instructor subscale. 

Research Question #6: Are there differences in student ratings of instruction across the different years of 
administration? 

An Analysis of Variance was used to assess if there were differences in the mean ratings over die seven year 
period from 1992 to 1999. The results are provided in Table 13. The results of this analysis support die stability 
of the SAI scores over time, as there were no significant differences in means over die seven years on any of the 
subscales or the total scale. 



DISCUSSION 

The faculty profile in the College of Education at ETSU seems to have remained fairly consistent in overall 
numbers from 1992 to 1999. However, the percentages in the different departments do seem to have shifted, at least 
on the basis of the number of reported ratings. The CUAI and the PEXS departments have lessened their 
proportions of total faculty ratings in the college, whereas the HDAL department increased its percentage. The 
ELPA department is still at about the same level. 

The percentage of ratings for males and females are similar, although ETSU has had an increase in the number 
of female faculty member ratings from 46.3% in 1992 to 53.2% in 1999. This trend is similar to the changing 
demographics noted nationally. ETSU has followed the trend of using an increasing percentage of part-time 
faculty, going from 17.7% of the total number of ratings in 1992 to 39.9% in 1999. If TAs are included, the percent 
has changed from 24.5% in 1992 to 48.3% in 1999. 

The results indicate that there are statistically significant differences in the ratings of faculty members of 
different ranks. As a whole, the instructor rank seemed to receive some of the highest ratings on the subscales, 
although most of the instructors were found in PEXS. The assistant professor rank had some of the lower ratings on 
the total scale, and adjunct/part-time faculty and associate professors were perceived similarly to full professors. At 
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least in our sample, part-time faculty were not perceived as offering less instructional quality. In fact, adjunct 
faculty were typically rated higher than associate or assistant professors, except in ELPA. The question from this 
study may be the reasons behind the assistant professors’ receiving lower ratings than some of the other ranks. 
Other factors may be affecting the quality of instruction received from this rank. This is material for further study. 

The lack of significant differences between the two main semesters and between genders suggests that school 
administrators should have some flexibility in scheduling courses. The time that a course is offered and the gender 
of the instructors should not affect the perceived quality of the instruction. The lack of differences in ratings across 
the years suggests a consistent level of instructional delivery over time. 

The differences in ratings between departments may be influenced by the relative number of graduate and 
undergraduate courses offered. Students in the ELPA department may be more sensitive to evaluation issues, given 
their area of emphasis, and rate courses more highly. In any case, the difference among departments represents a 
potential target for further study. 

SUMMARY AND CONCLUSIONS 



The data analysis produced some very interesting results. The trends in faculty demographics at ETSU seem to 
be similar to those at other institutions. It appears from the proportions of ratings that there has been an increase in 
the proportion of female faculty members. There also appears to be a much higher use of part-time faculty than in 
past years. The term in which a class was taken did not appear to affect the ratings of faculty. Neither the gender 
of the instructor nor the year of the evaluation demonstrated significant differences in the ratings. 

The subscale differences between instructors of different ranks were also interesting. Professors and 
instructors/lecturers tended to have higher scores than associate professors. This raises questions about the 
instructional activities of these two groups. Why do professors and instructors have similar high interest subscales? 
It could be that instructors have less committee and administrative work (than associate or assistant professors) and 
are therefore more able to concentrate on teaching, whereas professors have more experience than the other groups. 
Additional research would have to be done to explore the issue more thoroughly. In the literature of higher 
education it is sometimes implied that the quality of instruction suffers when large numbers of adjunct faculty are 
used. This was certainly not evident in these results. In our study, adjunct/part-time instructors did not receive 
lower rating scores than full-time faculty, except in ELPA. In many cases, adjunct faculty members were rated 
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higher than full-time faculty. Of the 38 instances where there were significant differences in the ratings of adjunct 
faculty and other groups (across the 6 scales), in 30 of those instances the adjunct faculty members were rated 
higher. 

This data set is a most illuminating one with which to work and could offer material for further study. The 
S AI instrument itself could be analyzed in more detail for validity, bias, and reliability. It should be noted that it 
uses the class mean as the unit of analysis as recommended by Cranton and Smith (1990). In general, students 
seemed fairly satisfied with courses taken in the College of Education. It could be that the small significant 
differences noted should not be “over-interpreted,” to use d’Apollonia and Abrami’s word of caution (1997). 

As shown in this data analysis, the instructor of a course does matter, as does the content of the course. 
However, what the student brings to the course is also important. Finding a good balance between what the student 
brings to a course, what an instructor brings to a course, and what the content itself offers appears to be the 
challenge. 
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TABLE 1 

DESCRIPTIVE PROFILE OF FACULTY MEMBERS IN THE COLLEGE OF EDUCATION 

BETWEEN 1992 AND 1999 
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Key: T= Total Sample, C= Curriculum & Instruction, E= Educational Leadership, H= Human Development and Learning, P= Physical Education, Exercise 
and Sports Science. 



TABLE 8 

COMPARISON OF SIGNIFICANCE OF RATINGS ON THE TOTAL SCALE 
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COMPARISON OF THE TOTAL AND SUBSCALE MEANS OF GRADUATE AND UNDERGRADUATE COURSES 
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COMPARISON OF THE TOTAL AND SUBSCALE MEANS OF SPRING AND FALL SEMESTERS 
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